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Abstract. We prove rigidity for hypersurfaces in the unit (n + 1)- 
sphere whose scalar curvature is bounded below by n(n — 1), without 
, ^ imposing constant scalar curvature nor constant mean curvature. The 

^ I lower bound n{n~l) is critical in the sense that some natural diflterential 

operators associated to the scalar curvature may be fully degenerate at 
geodesic points and cease to be elliptic. We overcome the difficulty by 
developing an approach to investigate the geometry of level sets of a 
height function, via new geometric inequalities. 

o 
q 

j2 1- Introduction 

^ Hypersurfaces in S'^^"^ of either constant scalar curvature or constant 

G mean curvature (including minimal hypersurfaces) have been extensively 

studied in the literature. For example, for constant scalar curvature, the 
pioneering work of Cheng- Yau ^ says that if a hypersurface M in has 
constant scalar curvature which is greater than or equal to n{n — 1) and non- 
negative sectional curvature, then M is either an re-sphere or a Riemannian 

Tj" product of a p-sphere and an (n — p)-sphere. They introduced a self-adjoint 

operator [5] which has been used by many other people to study hypersur- 
faces of constant scalar curvature, under various additional conditions (see, 

O for example, [DlIS]). 

In this paper, we consider hypersurfaces M in S"+^ whose scalar curva- 
ture R is bounded below by re(n — 1), namely, R > n[n — 1). We do not 

• w^ assume constant scalar curvature nor constant mean curvature. We would 

^ like to prove the rigidity of M under certain boundary condition. First, 

the boundary condition is necessary to derive rigidity results, because small 
spheres and their slight perturbation have sufficiently large scalar curva- 
ture. Furthermore, the lower bound n(n — 1) is critical because, by the 
Gauss equation, R > n{n — 1) implies that H'^ > \A\'^. Thus, the mean 
curvature H may change signs locally through the geodesic points, i.e., the 
points where the second fundamental form A vanishes. From the analytic as- 
pect, the geometric operators, such as Cheng- Yau's operator, the linearized 
scalar curvature operator, and the scalar curvature flow, are no longer glob- 
ally elliptic nor parabolic. Also, these operators are totally degenerate at 
the geodesic points. Therefore, the theory of maximum principle cannot be 
applied directly. (We remark that, if one assumes that M is contained in 
the hemisphere and that M has constant scalar curvature, then the desired 
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ellipticity automatically holds; and hence, M is a sphere by applying the 
Alexandrov reflection principle (see Korevaar pj]).) Rather than sticking 
to these geometric operators, we develop a different approach using some 
new geometric inequalities, motivated by our recent work |10j . 

Another important motivation of this paper comes from Min-Oo's con- 
jecture. Let M be an n-dimensional compact Riemannian manifold of scalar 
curvature R > n{n — 1) with boundary dM. Suppose that dM is isomet- 
ric to the unit sphere S*^"^ and is totally geodesic in M. The conjecture 
states that M is isometric to the hemisphere S". There have been many 
interesting results related to the conjecture. For example. Hang- Wang [71 [8] 
proved the conjecture under the condition that, either g is conformal to 
the standard sphere metric, or the Ricci curvature satisfies Ric > (n — l)g. 
By assuming positive Ricci curvature on M and an isoperimetric condition 
on dM, Eichmair [6] proved the conjecture for n = 3. Recently, Brendle- 
Marques-Neves [3j constructed counter-examples to Min-Oo's conjecture by 
using beautiful deformation techniques. For other recent results, we refer to 
the excellent survey by Brendle [2] and the references therein. 

In an early work [9], we proved Min-Oo's conjecture for M being a hy- 
persurface with boundary in either Euclidean space or hyperbolic space, by 
applying the strong maximum principle to the mean curvature operator. It 
is a natural continuation to study the case when M is in S"+^ However, 
the method in [9j does not apply to the spherical case, mainly due to the 
failure of ellipticity. Nevertheless, we are able to overcome the difficulty and 
obtain the following result. 

Theorem 1. Let M be a connected and embedded C"^^ hypersurface in S"+^ 
with boundary dM . Suppose that (M, dM) satisfies the following conditions: 

(1) The scalar curvature of M satisfies R > n{n — 1); 

(2) M is tangent to a great n-sphere S" at dM , and dM is the great 
(n — l)-sphere E^~^ . 

Then, M is the hemisphere . 

Theorem [T] confirms Min-Oo's conjecture when M is a hypersurface in 
gn+i^ under a weaker boundary condition. In fact, we prove the following 
stronger version, where dM is not assumed to be the great (n— l)-sphere (see 



Definition 4.5). We remark that the boundary condition below is necessary. 



in view of Example 6.2 



Theorem 2. Let M be a connected and embedded C^~^^ hypersurface in S"+^ 
with boundary dM. Suppose that (M, dM) satisfies the following conditions: 

(1) The scalar curvature of M satisfies R > n{n — 1); 

(2) M is tangent to a great n-hemisphere at dM . 

Then, M is a portion of the hemisphere S!j:. 

As a direct corollary, for closed (i.e, compact without boundary) hyper- 
surfaces in S""*"^, we have the following result. 
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Theorem 3. Let M be a closed, connected, and embedded C^+^ hypersurface 
in S""*"^ satisfying R > n(n— 1). If M is tangent to a great n-sphere at a great 
(n — l)-sphere. Then, M is the great n-sphere; namely, M is an equatorial 
sphere in S""*"^. 

We introduce a geometric inequality which quantifies geometry of the 
level sets in M of a height function. Although the maximum principle type 
argument fails to work on M, we are able to apply the maximum principle 
to the level sets. This relies on the geometric inequality which relates the 
scalar curvature and mean curvature of M in S"'"'"^ to the mean curvature 
of the level sets of a height function. The analogous situation occurs for 
hypersurfaces in Euclidean space with nonnegative scalar curvature. By the 
new technique, we have shown that if M is closed hypersurface in M""*"^ with 
nonnegative scalar curvature, then it must be mean convex [lOj . However, 
in the spherical case the geometric inequality is more involved. We deform 
a family of conic hypersurfaces to obtain the desired level set in M, where 
the inequality is utilized at full strength. We believe that this approach 
would have further applications in more general ambient manifolds. In the 
following theorem, we derive the geometric inequality for a large class of 
ambient spaces. 

Let (A^, g) be an n-dimensional Riemannian manifold, M a hypersur- 
face in the product manifold {N x M, g + dt'^), and u the unit normal vector 
field to M in (A X M,5f + dt^). Let S = {t = e} n M be a regular level 
hypersurface in M, and r] the unit normal to S in (A x {e},g). 

Theorem 4. Let M and S be given as above. Let A x M endow with a 
metric </>~^(g + dt'^), for a positive function cj) G C^{N x M). We denote 
by A and Aj^ the shape operators of M in (A x M, 0^^(gf + dt'^)) and S in 
(A X {e},4>~'^{-,e)g), respectively; and denote by H, Hy, the corresponding 
mean curvatures. Then, at any point ofTi, 

H[{u,7])HY + {n-l){iJ,dt)^t\ 

>1{H'- \A\') + ^^-^ [{u, r^)HY + {n- l){u, d^t] \ ^^'^^ 
where (f)t = d4>/dt, and (•,•) is with respect to the product metric g + dt 
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The equality in (1.1) holds at a point of S if and only if M and S satisfy 
the following conditions at the point: 

(i) S is umbilic in {N x {e},(f)~'^{-,e)g). Let k be the principal curvature. 

(ii) M C (A X M, has principal curvature {h',rj)K + {u,dt)(j)t with 
multiplicity at least n — 1. 

As an independent result, we obtain below a rigidity theorem for hyper- 
surfaces M in W^~^^ with nonnegative scalar curvature, and the boundary 
dM being planar, i.e., dM is contained in a hyperplane and M is tangent to 
the hyperplane at dM from the region enclosed by dM (see also Definition 



5.2). The boundary condition here is necessary, as shown in Example 6.1 
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Theorem 5. Let M be a compact, connected, and embedded (7""^^ hyper- 
surface in W^^^ with boundary dM. Suppose that dM is planar. If the 
scalar curvature of M is nonnegative, then M is contained in a hyperplane; 
namely, M is Euclidean flat. 

The rest of this paper is organized as fohows. The geometric inequahties 
are derived in Section [2] and Section [3} The rigidity results for hypersurfaces 
in spheres are proved in Section [4j while the result in Euclidean space is 
proved in Section [5| In Section |6j we provide two examples to indicate 
that the boundary conditions imposed in Theorem [2] and Theorem [5] are 
indispensable. 

Acknowledgement. In an earlier version of this paper, we proved a weaker 
version of Theorem [T] using the mean curvature flow. We would like to thank 
Gerhard Huisken and Tom Ilmanen for helpful discussions along that direc- 
tion. The first author is grateful to Panagiota Daskalopoulos for discussions 
and to the Albert Einstein Institute for their hospitality and generous sup- 
port. She is also partially supported by the NSF through DMS-1005560. 
The second author would like to thank Jianguo Cao and Brian Smyth for 
some conversations. 

2. Geometric inequalities on Product manifolds 

Let (A^, g) be a n-dimensional complete Riemannian manifold, and V 
the Levi~Civita connection of g. Consider the product manifold N x M. 
endowed with the product metric g + dt"^, which is also denoted by (•, •). 
Since (M, dt'^) is flat, by abuse of notation, we also use V to denote the Levi- 
Civita connection on the product manifold. Let / be a C'^ function over an 
open subset C N and {x^, . . . , x"} a coordinate system in a neighborhood 
of X G 17. We denote 

a _ f _ f i _ Jk f 



k=\ 



•^'i = Q ,M = ^^^^ foi" all i,j = l,...,n, 

and dn+i = d/dt. Then, for i = 1, . . . ,n, Cj = di + fidn+i is tangent to 
the graph of /, and {ei, . . . , Cn} forms a coordinate frame. Let v be the 
unit normal vector of the graph of / in A x M. The shape operator A^- 

with respect to v is defined by = X]fc=i 9^^{^j ^ek^j)^ h j = ■ ■ • ^n. 
By a direct calculation, the shape operator with respect to the upward unit 
normal vector is given by 

^ _ _rZ^\ 

Z^\y 1 I |Vf|2 j /I , \Y7f 



k=l 




(2.1) 



Vl + |V/|2 
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Let M be a hypersurface in the product manifold (A^ x M, (7 + dt'^) and 
ly the unit normal to M in x M. Consider the (nonempty) level set 
T, = M n {t = e}. By the Morse-Sard theorem (see [12] for example), for 
almost every e, S is a submanifold of M and also a submanifold in 
N X {e}. Denote by r/ the unit normal vector to S in {N x {e},^) and by 
Hy, the mean curvature of S with respect to ij. Also, we denote by cri{A) 
the trace of a matrix A. 

Lemma 2.1. Let T, = M Ci {t = e} for some regular value e. For a point 
p G S C M, let A be the shape operator of M in {N x 'K,g + dt'^) with respect 
to the unit normal v, and Aj] the shape operator of S in {N x {e}^g) with 
respect to rj. Then, there exists a coordinate chart (x^, . . . ,x") of p in M 
such that (x^, . . . ,x")|x; is a coordinate chart of p in S, and that 

{A\l) = {iy,7])As atp, 

where {A\l) stands for the (n — 1) x (n — 1) minor in the matrix A with the 
first row and first column deleted, and (•, •) is with respect to g + dt^ . In 
particular, taking the trace of the above identity, we have 

ai{A\l) = {v,vi)H^ at p. 

Proof. Without loss of generality, we can assume that (i/, r/) > at p, for 
otherwise we can just replace r] by —77. We divide the proof into the following 
two cases: 

Case 1: Suppose that {u^rf) < 1 at p. Then, M, locally near p = (x,e), 
is the graph of / over a domain ^ d N . Let v be the upward unit normal 
to M given by 



Y^f^di+ dn+l 



We can choose a coordinate system {x^, . . . , x"} near x so that at x, gij = 5ij, 
dgij/dx^ = 0, and 

/i = |V/|, /„ = for a = 2,...,n. 
By (2.1), we obtain at p that 

A'^ = -^"^ for aU 2 < a, /3 < n. 

On the other hand, because S near p is given by {x G : /(x) = e} and 
{i^, 7?) > at p, we have rj = — Ylk=i /^^fc/l^/l- Then, at p, 

(A^r, = V, (^) = ^ for an 2 < a, /3 < n. 

Since (i^, r/) = |V/|/ \/l + |V/p, we obtain 

iA\l) = {u,rj)Aj: at p. 
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Case 2: Suppose that (z^, ry) = 1 at p. That is, v = i] at p. Let 
{ei, . . . , Cn, Cn+i} be a local frame in a neighborhood of p in x R such that 
en+i = f, and that {ei, . . . , en} is a coordinate frame on M where ei|s = dt, 
{e2, . . . , e„} restricted to a coordinate frame on S. Then, the shape oper- 
ator of M in TV X M at p is A) = Y2=i d^'^di^: Ve^ej) for 1 < i, j < n. In 
particular, for a, /3 = 2, . . . , n, 

n 
7=2 

Observe that the shape operator of S in [N x {e},5() at p is 

n 

(^e)^ = a'^^aiv, Ve,e^) = for all Q, /3 = 2, . . . , n, 

7=2 

since u = r] at p. We finish the proof for Case 2. □ 

For a hypersurface M in [N x M, t^+dt^), the proposition below follows 
from the Gauss equation. 

Proposition 2.2. Let M be a hypersurface in {N xM.,g + dt^) with the 
induced metric g^^ . Let v he the unit normal vector to M. Denote by A the 
shape operator and by H the mean curvature with respect to v, respectively. 
Then, 

where v' = v— (z^, dtjdt in which (•, •) is with respect to g + dt^ , and R stands 
for the scalar curvature. 

Proof. Applying the Gauss equation to M in x R yields 

R{g + dt") = 2mCg^dA^, ^) + R{9^') -H^ + I^P- 

By the curvature formulas of a product metric (see, for example, \\.4\ p. 
89]), we have 

R{g + dt^) = R{g), mcg+at2{u,u) = Ricg{iy',iy'). 

□ 

Now we are able to prove the following geometric equality for product 
metrics: 

Theorem 2.3. Let {N,g) be an n-dimensional Riemannian manifold, M be 
a hypersurface in the product manifold [N xM., g + dt'^) , u the unit normal 
to M in N xM., and H, R{g^) the mean curvature and scalar curvature of 
M in N X respectively. Let T, = M (1 {t = e} for some regular value e, 
and r] the unit normal to T, in N x {e}. Then, at any point ofE, we have 

MHHj, > \R{g^') - \R{g) + {,^,r^fRicgiv,r]) 

n (2-2) 
+ 2(n^^-^)^^- 
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in which (•, •) is with respect to the metric g + dt^ , Hy, is the mean curvature 
of Ti in N X {e}, and R{g) and Ricg are the scalar curvature and Ricci 
curvature of{N,g), respectively. The equality in (2.2) holds at a point ofTj 
if and only if M and S satisfy the following conditions at the point: 

(i) S is umbilic in N x {e}. Denote by k the principal curvature. 

(ii) M d N xM. has principal curvature {i',r])K with multiplicity at least 
n — 1. 



Proof. The inequality (2.2) follows immediately from applying the linear 
algebraic identity (|10j. see also Pro posit ion 2.4 below) to Lemma 2.1 with 
2a2{A) = H'^ - and Proposition [2^ with u' = {u, r])r]. □ 



Proposition 2.4 (|ilOj). Let A = {oij) be a real n x n matrix with n > 2. 
Denote 

n n 

cri{A) = '^aii, ai{A\l) = '^aii, a2{A) = ^ {auajj - aijaji). 

i=l i=2 ^<i<j<n 

Then, we have 

ai{A)ai{A\l) = a2{A) + —^[ai{A\l)]^ + ^ a^a,-, 

i<«<i<" 



In particular, if aijOji > for all 1 < i < j < n, then 



(2.3) 



a,{A)ai{A\l) > a2{A) + ^^—^^[a,{A\l 

where "=" holds if and only if 022 = ••• = a^n and aijOji = for all 
1 < i < j < n. 

3. Geometric inequalities for conformal metrics 

In this section, we shall generalize the geometric inequality to the con- 
formal product metrics. Namely, we would like to prove Theorem |4j Let 
us first recall a general formula for the shape operator under conformal 
transformation (see, for example, [H p. 183]). 

Proposition 3.1. Let {M,g) be an m-dimensional manifold, M a hyper- 
surface in M , and n the unit normal vector to Ai with respect to g. Let 
{e2, . . . , Cm} be a local frame field in Ai, and A^- the shape operator of Ai 
with respect to g, /i, and {e2, . . . , Consider g = 4>~'^g for some positive 
function (p in J\f. Let A^j be the shape operator on A4 with respect to g, <j)fi, 
and {e2, . ■ ■ ,en}- Then, 

A) = ^A] + ^{(t))5i, 2<i,j <m. 

As a consequence, 

H = (t)H + {m-l)^i{<l)) (3.1) 
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where H and H denote the mean curvatures of M with respect to g and g, 
respectively. 

Let us recall the setting in Theorem |4j Let {N, g) be an n- dimensional 
Riemannian manifold, M a hypersurface in the product manifold {N x 
M, 5 + dt^), and v the unit normal vector field to M in x M. Let S = {t = 
e} n M be a regular level hypersurface in M, and the unit normal to S in 
(A^ X {e},g). We denote by A and A^, the shape operators of M C iV x M 
and A'^ X {e} with respect to v and r/, respectively. Then, (Ti{A) = H and 
(^i{Ay;) = H-£. Let X M endow with a metric (j)~'^{g + dt'^), for a positive 
function (j) G C^{N x M). We denote by A and Aj^ the shape operators of 
M in (A^ X M, ^"^(^ + dt'^)) and S in (A^ x {e}, e)g), respectively; and 

denote by H, Hs the corresponding mean curvatures. 

To prove Theorem |4j it suffices to establish the following lemma, similar 
to Lemma 2.1 Then, Theorem [4] follows immediately by substituting (3.2) 



into Proposition 2.4 



Lemma 3.2. With the notation in Theorem^ for any point p £ T, C M, 
there exists a coordinate chart {x"^, . . . , x") of p in M such that (x^, . . . , x") 
is restricted to he a coordinate chart of p in S, and that 

{A\l) = {u,ri)A^ + {v,dt)(t>tIn-i atp, 

where (•, •) is with respect to g + dt^ , {A\l) is the matrix obtained by deleting 
the first row and first column of A, and In-i denotes the (n — 1) x (n — 1) 
identity matrix. In particular, 



Proof. By Lemma 
M, such that (x^, 
and that 



ai{A\l) = {u,ri)Hj^ + [n-l){u,dt)ct>t at p. (3.2) 

there exists a coordinate chart (x^, . . . , x") near p in 
x") is restricted to be a coordinate chart of p in S, 



2.1 



{A\l) = {v,r,)Aj:. 



Fix this coordinate chart. Applying Lemma 3.1 to (x^, . . . 

= (t)A^ + ri{(t))In~i. 
By applying Lemma 3.1 again to (x^, . . . , x"), we obtain 

{A\l)=^{A\l)+v{(fyIn-l 



,x" 



(3.3) 
Is yields that 
(3.4) 



(z^, ri)ct)Aj: + u{(t))I, 



n—li 



by (3.3) 



{u,ri)A^ + [u{<i)) 
{v,rf)A^ + {v,dt) 



kln-l- 



by (3.4) 



□ 



As corollaries of Theorem |4j we obtain the following results for hyper- 
surfaces in Euclidean space {W^^^^go), and hypersurfaces in a unit sphere 
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(M"+\g5), where 

gs = r'90, and <P = ^^^^^f^. 

Corollary 3.3. Let M be a hypersurface in {W^~^^,go). Denote by S the 
regular level set of a height function, say S = M n {x""*"-^ = e} for a regular 
value e. Let v and rj be the unit normal vectors to M in (M""''^,(7o) md S 
in (M" X {e},gio); respectively. Then 

go{i^,v)HHj: > | + ^^^-^^go{v,r^fHl, 

where the equality holds at a point of S if and only if (M, S) satisfies the 
following conditions at the point: 

(i) S is umbilic in (M" x {e},(7o)- Denote by k the principal curvature. 

(ii) M has principal curvature go{u,r])K in (W^^^,gQ) with multiplicity at 
least n — 1 . 



Corollary 3.3 recovers a result in our previous work \10\ Theorem 2.2], 
which is employed to prove Lemma 5.1 The following corollary will be used 
to prove Lemma |4.3[ 



Corollary 3.4. Let M be a hypersurface in (W^'^^ , gs). Denote by S the 
regular level set of a height function, say S = M n {x"~^^ = e} for a regular 
value e. Let v and r\ be the unit normal vectors to M in {W^~^^,go) and S 
in (M" X {e},go), respectively. Then 

H [go{iy,v)H^ + {n - l)<?o(i^, 5„+i)x"+i] 

> - n{n - 1)] + [go{u, r])H^ + (n - l)go{u, ' , 

where the equality holds at a point of S if and only if (M, S) satisfies the 
following conditions at the point: 

(i) T, is umbilic in (R" x {e},5s). Denote by k the principal curvature. 

(ii) M has principal curvature go{i^,r])K + go{i', dn+i)(l)n+i in (M"'*"^,5'5) 
with multiplicity at least n — 1. 



4. Rigidity in spheres 

In this section, we study a compact hypersurface M in satisfying 
R > n{n — 1) with nonempty boundary dM. Here we denote by S'^ the 
fe-dimensional unit sphere, and by the closed fc-dimensional hemisphere. 
We will make use of the following quasi- linear elliptic operator H{u). 
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Definition 4.1. Let W be an open subset in M" and u G C^iW). Define 
the operator 



l + \x\-^+U^ S-^ ( UiUj \ Uij 

= — -2 — E - rr^j Tfrm 

1,3 = 1 • • ' V • \ \ 

-.{u — X ■ Du). 



n 



a/1 + 

By (3.1 ), -ff (n) is the mean curvature of the graph of u in (M""*""^, ^i^) with 



respect to the upward unit normal vector 

. {-Du,l) 



2 



Here {gs)ij = <p-^5,j and = [1 + Y.tli^"?]/'^- 

Proposition 4.2. Let W he an open subset in M", p € dW . Denote by 
B{p) an open ball centered at p in M". Let u G C'^{B{p) n W) satisfy that 
= 0, \Du\ = on B{p) n // H{u) > on B{p) n T^, then 

{x e n W : u{x) > 0} / 0. 

Proof. Suppose on the contrary u < on B{p) W . If u = somewhere 
in B{p) n W, then u attains a local maximum, which contradicts H{u) > 0. 
Thus, u is strictly negative on B{p) n W. Furthermore, because B{p) n W 
is open, there is a smaller open ball B contained in B{p) CiW so that dB 
touches q G B{p) D dW . Namely, dW satisfies an interior sphere condition 
at q. Because n < and H{u) > on S and u{q) = 0, we conclude that 
\Du\{q) 7^ 0, by the Hopf boundary point lemma (see, for example, [9| p. 
359]). This leads a contradiction. □ 

Denote by Ba C the open ball of radius a centered at the origin, and 
denote Sa = dBa- 

Lemma 4.3. Let W <Z Bi be an open set in M" . Suppose that W H (-Bi \ 
Ba) / for some < a < 1. Let u £ C'^ (w7Ujh\Ba)) satisfy that 
ti = 0, \Du\ = on dW D {Bi \ Ba). If the scalar curvature R of the graph 
of u in {MP'^'^,gs) satisfies R > n{n — 1), then H{u) = at some points in 
W n {Bi \ Ba). As a consequence, LL{u) = at a sequence of points in W 
which converges to some point in dW . 

Proof. Suppose, on the contrary, that H{u) is never zero over W r\{Bi\Ba). 
We may assume that LL{u) > in n [Bi \ Ba)., for otherwise, replace u 
by —u. For a constant A, consider 

-ipxix) = A(l - for ah x ^ Bi\ Ba- 

Because u = Q and \Du\ = on dW r\ {Bi \ Ba), we can choose a sufficiently 
large A so that 

ijx>u onWn{Bi\Ba). 
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Let US consider a smaller domain W Ci {Bi\ Ba') for some a < a' < 1. We 
continuously decrease A until, for the first time, iPx{xq) = u{xo) at some 
xq £ W D {Bi \ Ba'). By Proposition 4.2, we have u{xo) > and A > 0. 
Furthermore, we have 

\Du\ > \Dru\ > {Dri'xl = A at xq, (4.1) 

where Dr = Y.iix'' /\x\)di. 

Let e = u{xo), and consider the level set 

:= {{x, e) G M"+^ : x e n (Si \ Ba), u{x) = e}. 

Because \Du\{xo) 7^ 0, is smooth near xq. Let r/ = (j)Du/\Du\. Then, 
is a unit normal to in ({x"+^ = e},gs) near xq. Denote by Hy:^ the 
mean curvature of in {x""'"^ = e} with respect to ??. Since H{u) > and 
i? > n(n — 1), by Corollary 3.4 we obtain 

\Du\ (n — l)u 

< , ' ' Hs, + ; ^ at Xq. 

- ^/l + \Du\^ ' VT+\D^ 

By above inequality and (4.1), at xq, 

H^. < <{n- 1)-^ = (n - 1)(1 - Ixol). (4.2) 

\L)u\ \Dripx\ 

On the other hand, note that T,' := Si^^i x {e} is the level set of ipx and is 
tangent to He at xq. By taking trace of (3.4) and computing directly, the 
mean curvature Hy:' of S' in ({x""*"^ = e},gs) with respect to the inward 
unit normal vector is 

/n - 1\ 1 + - |xoP 

By 



By the construction, we have at xq that 

Hy,_ > Hy' > (n-l)(l-|xo|). 



This contradicts with (4.2). □ 

For the rest of the section, let M be a compact hypersurface in S"'^^ with 
nonempty boundary dM. Denote by A, H, and R, the second fundamental 
form, mean curvature, and scalar curvature of M in S""*"^, respectively. Let 
int(M) be the set of interior points in M, i.e., int(M) = M \ dM. The set 
Mq of interior geodesic points is given by 

Mo = {p£ int(M) : A{p) = 0}. (4.3) 

The following lemma gives a useful characterization of Mq. This is the only 
place that requires M to be C""*"^ rather than C^. The proof is parallel to 
the Euclidean case (see, for example, [H] and [10]). 



Lemma 4.4. Suppose that Mq / 0. Let Mq be a connected component of 
Mq. Then Mq is contained in a great n-sphere o/S"'"'"^ which is tangent 
to M at every point of Mq . 
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Proof. Consider the Gauss map u : int(M) — )• S**. Since M is of C""*""^, the 
Gauss map is of C". Note that the Gauss map v has rank zero at any point 
of Mq. We can then apply a theorem of Sard \W\ p. 888, Theorem 6.1], 
to obtain that image I'iMo) has Hausdorff dimension zero in S"". It follows 
that z^(Mo) is totally disconnected in S"'. Then ^{Mq) consists of a single 
point, denoted by uq. 

It remains to show that Mq lies in a great sphere which is orthogonal 
to vq. Let S Mq. Consider the stereographic projection from S"+^ to 
]^n+i i\YSii pq is mapped to the origin of M'^+-^. Let {V; yi, . . . , Un) be local 
coordinates centered at in M-, and define 

^{y) = gs{vo,x{y) - x(0)), for each y = (yi, . . . € 1/. 

Here x = (xi, . . . , Xn+i) is the coordinates in M'"+-'^ such that x(0) = 0, and 
55 is the spherical metric on M*^"^^. Then, the function (p S C^^^iV), and 
by our construction, 

M'Qr\Vc{yeV\ |^(y) = 0, i = l,...,n]. 
oyi 

It follows from a theorem of A. P. Morse jT3t p. 70, Theorem 4.4] that (p is 
a constant on Mq n thus, 99 = on Mq n y. Since Mq is connected, 

55(1^0, 2;(y) - x(0)) = for all y ^V, 

that is, Mq lies in the hyperplane through the origin which is orthogonal to 
vq. Therefore, Mq lies in a great n-sphere orthogonal to vq. □ 

Before we proceed to prove Theorem [2j let us make precise some termi- 
nology. 

Definition 4.5. For a hypersurface M C S""*"^ with boundary 9M, we 
say that M is tangent to a great n-sphere S" C S""*"-^ at dM, if dM lies in 
S", and a neighborhood of dM in M can be represented as the graph of a 
function u over an open subset V in S*^ with li = 0, = on dM. 
Furthermore, we say that M is tangent to a hemisphere S" C S""^^ dM, 
if 5M C and the open subset V given above lies in the component of 
E>1 \ dM which does not contain dS^. (In the case that dM = SSIf:, V is 
simply an open subset in the open hemisphere). 

Remark 4.6. Equivalently, a hypersurface M C S"'^^ is tangent to a 
hemisphere C S""*"^ at dM if there exists a stereographic projection from 
to M""'"-'^ such that the following properties hold: 

(1) is mapped onto the closed unit ball Bi in {x"^^ = 0}, and dM 
is mapped onto a closed subset in Bi ; 

(2) a neighborhood of dM in M can be represented as the graph of a 
function u £ C'^{V) over an open subset V C {x"^~^^ = 0} such that 
u = and |-Du| = on dM (here we identify dM with its image); 

(3) V is contained in the region enclosed by dM. 

We are now ready to prove Theorem [2] 
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Proof of Theorem [2I By Lemma 4.4 it is sufficient to sliow tliat M = Mq, 



where Mq is given by ( |4.3[ ). Since R > n{n — 1), by the Gauss equation, 

= \A\^ + R-n{n-l)> \A\^ . 

Thus, Mo = {p e int(M) : H{p) = 0}. We would Uke to show that H = ^ 
on M. 

First, we claim that Mq n dM 7^ 0. Since M is tangent to a hemisphere 



at dM, by Remark 4.6 there exists a stereographic projection and an 
open subset V in Bi C {x""*"^ = 0} such that a neighborhood of dM in M 
is the graph of a C""*"^ function u over 1/ in (M""'"^, gis) and that u = and 



\Du\ = on dM. Applying Lemma 4.3 with W = V proves the claim. 



Next, for any connected component Mq of Mq such that Mq n dM 7^ 



by Lemma 4.4, Mq lies entirely in {x^^^ = 0}. We denote by M^: the union 
of all such components Mq together with dM. Then, M* is a closed and 
connected subset in {x""*"^ = 0}. We want to show that M^ = M. Suppose 
not. Then, there exists a bounded open set W C {x^^^ = 0} enclosed 
by M=K, such that H is never zero over a neighborhood of dW intersecting 
W. Furthermore, note that W C Bi, because of Remark |4.6| ([s]) and the 
fact that M has no self-intersection. On the other hand, observe that M 
is tangent to {x"""*"^ = 0} at dW. Thus, M near dW can be represented 
by the graph of a C"^^ function v over an open subset in W, and v = 



and \Dv\ = on dW. Applying Lemma 4.3 yields a contradiction. Hence, 



M = M^. □ 

5. Rigidity in Euclidean space 

In this section, we shall prove Theorem [5} Throughout this section, we 
denote by M a connected, embedded, and orientable (7"+^ hypersurface in 
M"+i with nonempty boundary dM, unless otherwise indicated. We first 
recall the lemma about the Euclidean mean curvature operator that we 
established earlier in |lDj. Let / be a function defined over an open set 
in W^. The upward unit normal vector of the graph of / in (M"+^, go) is 

The mean curvature operator is defined by 

n 



Lemma 5.1 ([lOj). Let W he a hounded open suhset in R" and U he an open 
neighborhood of dW in R"". Suppose f G C'^{UnW) and / = 0, \Df\ = 
on dW nU . If the scalar curvature of the graph of f is nonnegative, then 
H{f) = some points in U W . □ 

Definition 5.2. For a hypersurface M C M"^^ with boundary dM, we 
say that dM is planar if the following conditions are satisfied: 
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(1) dM is contained in a hyperplane; 

(2) a neighborhood of dM in M can be represented as the graph of a 
function / over a domain 0, in the hyperplane, satisfying that / = 
and \Df \ = on dM; 

(3) ri is contained in the region enclosed by dM. 

We recall the following characterization of the set of geodesic points of 
complete hypersurfaces, due to Sacksteder [15] (see also [TO]). For our proof 
of Theorem [Sj this is the only place that requires the hypersurface M to be 
C"'"'"^, instead of C^. We denote the set of interior geodesic points by 

Mo = {pe int(M) : A{p) = 0}. 

Lemma 5.3. Let M be a C^+^ hypersurface in W^^^ with boundary dM . 
Let Mo be the set of interior geodesic points defined above. Let Mq be a 
connected component of Mq. Then Mq lies in a hyperplane which is tangent 
to M at every point in Mq. □ 

We now prove Theorem [Sj 

Proof of Theorem [5j By the Gauss equation and the assumption R> 0, 

H'^ = \A\'^ + R> \A\^. 

That is, Mo = {pe int(M) : H{p) = 0}. 

Let M=K be a connected component of Mq U dM such that M=k n dM / 0. 



By Lemma 5.3 and the condition that dM is planar, M^, C {x"^ = 0}. 
It suffices to prove M^, = M. By Definition 5.2 ([3]), M=k lies in the closed 



subset in {x^~^^ = 0} enclosed by dM. Suppose that M^ / M. Then, there 
exists a bounded open set W C {x""*"^ = 0} enclosed by M^, and H is never 
zero over a neighborhood of dW intersecting with W. On the other hand. 



observe that M is tangent to {x"^^ = 0} at M^. Applying Lemma 5.1 yields 
that H must vanish at points in W which are arbitrary close to dW. This 
is a contradiction. Thus, M^: = M, and M is contained in {x"^^ = 0}- ^ 

6. Examples 

In this section, we shall present two examples to demonstrate that the 
boundary conditions in Theorem [2] and Theorem [5] are indispensable. Thus, 
the results are optimal in this sense. 



Let us first consider Theorem 



where dM is assumed to be planar (see 



Definition 5.2). In particular, by (31) in Definition 5.2 we require that M is 
tangent to dM from the bounded region enclosed by dM. This is necessary, 
for, in the following example, M has nonnegative scalar curvature but is 
tangent to its boundary dM from the region outside of dM. 

Example 6.1. Consider the surface in Euclidean space (M^,go): 

M = {(x,y,z) G I 7x2Ty2-/(z) = 0}, 

where 

f(z) = (a/z + l)\/l - z2 foranO<z<l. 
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Figure 1 . The left figure indicates the graph of the function 
r = f{z) in Example |6.1[ The right figure indicates the 
graphs of z = ■u(r) and z = v{r) in Example 6.2, The surfaces 
are obtained by rotating the curves about the z-axis. 



Clearly, the surface M is obtained by rotating the curve f{z) about the 
z-axis. Note that M has nonnegative Gauss curvature, because f{z) is 
concave. Furthermore, M is smooth with boundary 

dM = {{x, y, 0) G I + y2 = 1}. 

It is readily to check that dM satisfies all conditions except ([s]) in Defini- 
tion 5.2 since M is tangent to the plane {z = 0} at dM from the region 
outside of dM. □ 

Next, let us consider Theorem [2j where M is assumed to be tangent to 
a great n-hemisphere at dM (see Definition 4.5 and Remark 4.6). Similarly 
to the Euclidean case, we remark that condition ([s]) in Remark 4.6 cannot 
be removed, in view of the following example. 

Example 6.2. Consider with the spherical metric gs, i.e., 

(l + a;2 + y2 + ^2)2' 

where go is the Euclidean metric and (x, y, z) are the Cartesian coordinates. 
We denote r = Y^x2~+~y2_ < a < 1 be a constant. Define 



1 / r 
n(r) = -2Vl-r - + 2^1- a + 



for all a < r < 1; and define 



v{r) = n(l) + \/l - r2 = + \/l - r^, for all < r < 1. 

Let M be the surface which is the union of graph of u over [a, 1] and the graph 
of V over [0, 1]. Then, we claim that M is a surface of scalar curvature 
> 2 in (M^, (75) (where R = 2 holds at and only at the boundary points). 
It is straightforward to check that 

dM = {(x, y, 0) G I x^ + y2 = < 1} 
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satisfies all the conditions in Remark |4.6| except ([s]). 

It suffices to show the claim. Note that the graph of is a portion of 
the unit 2-sphere centered at {r,z) = (0, \/ (1 — o)/2). With respect to the 
upward unit normal, the graph of v has principal curvatures 



Kl = K2 



HI)? 



1 



2 4 

Hence, R = 2 + 2kiK2 > 2 on the graph of v. For the function n, obviously 
we have u G C°°([a, 1)) n C''([a, 1]). In addition, u satisfies the following 
properties: 

u{a) = 0, u'{a) = 0, u'{r) > for aU a < r < 1, and (6.1) 

u"{r) > u[l + iuf], foraUa<r<l. (6.2) 

With respect to the upward unit normal, the principle curvatures of the 
graph of u are given by 

1 + + u" 



1 
1 

1 



ru + 



ru + 



2 1 + (m')^ 
1 + + u' 



u + u' 



Since r < 1, by (6.1) we have A2 > 0. Applying (6.2) to Ai yields that 
, ± -r (x -r 

u 



ru H 



u 



1 + (n')2 



> u — ru + 



1 + + 



-u 



u + u 



,u^ + (r - 1)^ 



> 0. 



2 2 
It follows that Ai > 0. Therefore, i? = 2 + 2A1A2 > 2 on the graph of u, 
where = 2 if and only if r = a. Obviously M is smooth at the interior 
points of graph u and graph v. It is elementary to verify that M is of at 
the intersection curve (r, z) = (1, \J(\ — a)/2) of the two graphs. Thus, the 
claim is proved. 

Now, because i? > 2 in a neighborhood of the intersection curve, we can 
perturb M locally near the intersection curve to get a smooth surface M in 
(M3,c/5) with R>2. Also, M is identical to M away from a neighborhood 
of the intersection curve. □ 
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